Portable electronic device for mixed reality headset

ABSTRACT

A portable electronic device is used with a mixed reality headset. The MR headset includes a lens and a holder that retains the portable electronic device in a defined orientation relative to the lens. The portable electronic device includes a camera, a display device, and a processor. The camera outputs video frames and is arranged to view the lens. The display device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user&#39;s eyes and the camera. The processor performs operations that include processing the video frames from the camera to identify locations of at least one real-world feature, displaying information on the display device, and controlling at least one of the processing of the video frames and the displaying of the information on the display device to at least partially reduce occurrence in the video frames of the displayed information.

TECHNICAL FIELD

The present disclosure relates to a portable electronic device for amixed reality (MR) headset, a method by a portable electronic device foran MR headset, and a corresponding computer program product.

BACKGROUND

Mixed Reality (MR) headsets will be an increasing part of future deviceecosystem used by society for interpersonal communication, gaming andother applications. MR headsets augment what a user sees in the realworld with computer generated information (e.g., graphical objects,text, indicia, etc.) having a pose that can make the information appearto the user as if it exists in the real world. MR headsets can enable auser to interact in a seemingly real or physical way using specialelectronic equipment, such as a haptic device that tracks movement ofthe user and which may provide haptic feedback to the user.

Smartphone or other portable electronic device-based MR headsets arebecoming increasing popular, with several startups launching new devicesin 2017 and 2018 called Mira Prism, Holokit, Ghost AR, Lenovo Mirage AR.These headsets are relatively inexpensive and can be used withsmartphones by Apple, Samsung, Google, and others. For example,developers for the Ghost and Holokit devices can utilize Google's andApple's MR software development kit (SDK) tools, called ARCore andARKit, respectively, to perform object recognition and 6degree-of-freedom (DOF) localization.

MR headsets can generally be characterized into two categories, amirror-less headset (e.g. Mira Prism type headset depicted in FIGS. 1and 2 below) and a mirror-based headset (e.g. Holokit and Ghost headsetsdepicted in FIG. 3). For a mirror-less headset, light emitted by adisplay device of a portable electronic device 120 (e.g., smartphone) isprojected toward a lens 110 (e.g., along path 200 a). The lens 100 is atleast partially transparent (i.e., see-through) to allow ambient lightto pass through for viewing by a user who is wearing the MR headsetwhile also being configured to reflect at least part of the light thatis emitted by the display device backward for viewing by the user. Thelens 110 may or may not have light concentrating or light dispersingeffects (e.g., magnifying light from real-world features). In FIGS. 1and 2, the lens 110 reflects the light toward the user's eyes (e.g.,along path 200 b, predominantly by virtue of specular reflection) andtoward a camera 122 of the device 120 (e.g., along path 200 c, by virtueof diffusive reflection). In contrast, for a mirror-based headset(illustrated in FIG. 3), the light emitted by the display device of theportable electronic device 120 is projected (e.g., along path 330 a)toward a mirror 320 which reflects light toward a reflective see-throughlens 310 (e.g., along path 330 b) for reflection which can be directlyor indirectly toward a user's eyes (e.g., along path 330 c,predominantly by virtue of specular reflection).

Some mirror-based headsets configure a rear facing (back) camera of theportable electronic device 120 to have a field-of-view of the world,which is beneficial because the rear camera can be configured to providebetter quality video than a front camera. For example, current versionsof the ARCore or ARKit SDKs can only operate effectively using the rearcamera due to its higher quality imaging relative to the front camera.Some mirror-based headsets enable more flexible positioning of theportable electronic device than can be allowed for mirror-less headsetsdue constrains of the relative positioning between the see-through lensof the headset and the display device of the portable electronic device.

An MR system can operate by having a display device of the portableelectronic device project information and other virtual content towardthe headset lens (mirror-less design) or the headset mirror(mirror-based design). Video frames from the camera are processed toidentify real-world features, such as tracking markers or otherreal-world objects, which can be used for positioning (e.g. tointerrelate real-world and virtual-world (i.e., MR headset reference)coordinate systems), feature recognition, and/or for streaming video toa remote viewer (e.g. for learning/teaching MR applications enabling aremote expert to view video captured by the technician's MR headset andprovide responsive guidance).

Because the headset lens or mirror serves two purposes, for reflectinginformation projected by display toward the user's eyes and for enablingthe user to view the real-world therethrough, the projected informationmay interfere with the user's ability to view the real-world.

SUMMARY

Some embodiments disclosed herein are directed to a portable electronicdevice for use with a mixed reality (MR) headset. The MR headsetincludes a lens through which a user wearing the MR headset can viewreal-world features. The MR headset also includes a holder that isconfigured to releasably retain the portable electronic device in adefined orientation relative to the lens. The portable electronic deviceincludes a camera, a display device, and a processor. The camera isconfigured to output video frames and arranged by the holder to view atleast a portion of the lens of the MR headset. The display device isarranged to display information that is projected on the lens forreflection directly or indirectly toward the user's eyes and the cameraof the MR headset. The processor is operationally connected to thecamera and to the display device, and configured to perform operations.The operations include processing the video frames from the camera toidentify locations of at least one real-world feature, displayinginformation at locations on the display device, and controlling at leastone of the processing of the video frames and the displaying of theinformation on the display device to at least partially reduceoccurrence in the video frames of the information that is displayed bythe display device.

Some potential advantages of these operations include that theinformation which is projected by the display device onto the lens iscontrolled to occur less often, if at all, in the video frames. Thevideo frames can thereby more accurately capture images of thereal-world and, thereby, enable more robust identification of where thereal-world features are present within the video frames and, therefrom,where the real-world features are located relative to the MR headset.

In some further embodiments, the processor controls a frequency at whichthe display devices repetitively refresh displayed information and/or afrequency which the video frames from the camera are processed toidentify locations of the real-world feature, based on whether a displayprioritization rule is satisfied or whether a camera videoprioritization rule is satisfied. Alternatively or additionally, theprocessor may operate to repetitively alternate between displaying theinformation on the display device and ceasing display of the informationon the display device, where the processing of the video frames iscontrolled to occur during first time durations while the information isnot displayed on the display device and controlled to not occur duringsecond time durations while the information is displayed on the displaydevice. These operations may further improve the quality at whichinformation is displayed on the display device and/or improve imagequality of the video frames from the camera.

Some other embodiments are directed to a corresponding method by aportable electronic device for use with an MR headset. The MR headsethas a lens through which a user who is wearing the MR headset can viewreal-world features. The MR headset also includes a holder that isconfigured to releasably retain the portable electronic device in adefined orientation relative to the lens. A display device of theportable electronic device is arranged to display information that isprojected on the lens for reflection directly or indirectly toward theuser's eyes and a camera of the portable electronic device. The methodincludes processing the video frames from the camera of the portableelectronic device to identify locations of at least one real-worldfeature, displaying information at locations on the display device ofthe portable electronic device, and controlling at least one of theprocessing of the video frames and the displaying of the information onthe display device to at least partially reduce occurrence in the videoframes of the information that is displayed by the display device.

Still some other embodiments are directed to a corresponding computerprogram product for a portable electronic device that is used with an MRheadset having a lens and a holder for the portable electronic device. Adisplay device of the portable electronic device is arranged to displayinformation that is projected on the lens for reflection directly orindirectly toward the user's eyes and a camera of the portableelectronic device. The computer program product includes anon-transitory computer readable medium that stores program code whichis executable by a processor of the portable electronic device toperform operations. The operations include processing the video framesfrom the camera of the portable electronic device to identify locationsof at least one real-world feature, displaying information at locationson the display device of the portable electronic device, and controllingat least one of the processing of the video frames and the displaying ofthe information on the display device to at least partially reduceoccurrence in the video frames of the information that is displayed bythe display device.

Other portable electronic devices and corresponding methods and computerprogram products according to embodiments will be or become apparent toone with skill in the art upon review of the following drawings anddetailed description. It is intended that all such additional portableelectronic devices, methods, and computer program products be includedwithin this description and protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example andare not limited by the accompanying drawings. In the drawings:

FIG. 1 illustrates a mixed-reality (MR) system that includes an MRheadset that holds a portable electronic device which is configured tooperate in accordance with some embodiments of the present disclosure;

FIG. 2 illustrates a side view of the MR system of FIG. 1;

FIG. 3 illustrates another embodiment of a mixed-reality (MR) systemwith a portable electronic device having a display device that projectsimages which are reflected directly or indirectly toward a user andtoward a camera in accordance with some embodiments of the presentdisclosure;

FIGS. 4-12 are flowcharts of operations that can be performed by aportable electronic device to operate in accordance with someembodiments of the present disclosure; and

FIG. 13 is a block diagram of components of a portable electronic devicethat are configured to operate in accordance with some other embodimentsof the present disclosure.

DETAILED DESCRIPTION

Inventive concepts will now be described more fully hereinafter withreference to the accompanying drawings, in which examples of embodimentsof inventive concepts are shown. Inventive concepts may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein. Rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of various present inventive concepts to thoseskilled in the art. It should also be noted that these embodiments arenot mutually exclusive. Components from one embodiment may be tacitlyassumed to be present/used in another embodiment.

FIG. 1 illustrates an MR system that includes an MR headset 100 thatholds a portable electronic device 120 which is configured to operate inaccordance with some embodiments of the present disclosure. Referring toFIG. 1, the MR headset 100 includes a lens 110 through which a user whois wearing the MR headset can view real-world features and furtherincludes a holder 121 that is configured to releasably retain theportable electronic device 120 in a defined orientation relative to thelens 110.

The portable electronic device 120 includes a camera 122, a displaydevice (1240 in FIG. 13), and a processor (1206 in FIG. 13). The camera122 is configured to output video frames and arranged by the holder 121to view at least a portion of the lens 110 of the MR headset 100. Thedisplay device 1240 is arranged to display information that is projectedon the lens 110 for reflection directly or indirectly toward the user'seyes, i.e., while wearing the MR headset 100, and the camera 122 of theportable electronic device 120. Although not shown, intervening mirrorsmay be positioned between the lens 110 and the user's eyes and/or thecamera 122 and, hence the light may be reflected directly or indirectlytoward the user's eyes and/or the camera 122. The portable electronicdevice 120 can include, but is not limited to, a smartphone, a palmtopcomputer, a tablet computer, or other computing device. As will beexplained in further detail below, various operations that are describedbelow as being performed by the portable electronic device 120 mayadditionally or alternatively be performed by another electronic devicethat is communicatively connected to the portable electronic device 120.

FIG. 2 illustrates a side view of the MR system of FIG. 1. Referring toFIG. 2, the display device 1240 of the portable electronic device 120displays information that is projected along path 200 a onto the lens110 for reflection backward along path 200 b toward the user's eyes andalong path 200 c toward the camera 122.

FIGS. 4-12 are flowcharts of operations that can be performed by aportable electronic device in accordance with some embodiments of thepresent disclosure.

Referring to FIGS. 2 and 4, the processor 1206 is operationallyconnected to the camera 122 and to the display device 1240. Theprocessor 1206 performs operations that include processing 400 the videoframes from the camera 122 to identify locations of at least onereal-world feature. The processor 1206 also displays 402 information atlocations on the display device 1240. As used herein, the term“information” may be graphical objects, text, indicia, etc. that isdisplayed on the display device 1240.

If various operational embodiments that are disclosed herein were notused by the portable electronic device 120, the video frames output bythe camera 122 would contain the information that is displayed by thedisplay device 1240 for the purpose of being viewed by the user. Thedisplayed information could appear in the video frames overlappingimages of the real-world, and may thereby obscure at least part ofvarious real-world features that the portable electronic device 120 isattempting to identify within the images of the video frames. Anundesirable consequence is that the portable electronic device 120 mayoperate less accurately or be prevented from operating to identifyreal-world features in the video frames.

However, these problems may be reduced or overcome by variousembodiments disclosed herein. In accordance with some embodiments, theprocessor 1206 controls 404 at least one of the processing of the videoframes and the displaying of the information on the display device 1240to at least partially reduce occurrence in the video frames of theinformation that is displayed by the display device 1240, which may bereflected by the lens 110 toward the camera 122. The description hereinof the processor 1206 of the portable electronic device 120 beingconfigured to control “at least one of” the two recited operations is tobe understood to mean that the processor 1206 may be configured tocontrol processing of the video frames without being configured tocontrol the displaying of the information on the display device 1240 inorder to at least partially reduce occurrence in the video frames of thedisplayed information, alternatively the processor 1206 may beconfigured to control the displaying of the information on the displaydevice 1240 without being configured in order to control the processingof the video frames to at least partially reduce occurrence in the videoframes of the displayed information, or still alternatively theprocessor 1206 may be configured to control processing of the videoframes and further configured to control the displaying of theinformation on the display device 1240 in order to at least partiallyreduce occurrence in the video frames of the displayed information.

Although the operations of FIG. 4 have been described in the context ofthe mirror-less MR headset 100 of FIGS. 1 and 2, the operations of FIGS.4-12 may also be performed by portable electronic devices which are usedwith mirror-based MR headsets. An example mirror-based MR headset 300 isshown in FIG. 3, with the head-worn assembly not shown for simplicity,which may be used with the portable electronic device 120. Referring toFIG. 3, the display device 1240 of the portable electronic device 120displays information that is projected along path 330 a onto the mirror320 for reflection along path 330 d toward the camera 122 and along path330 b toward the lens 310 and, optionally, through an intermediary lens312. The lens 310 reflects the incident information along path 330 ctoward the user's eyes, i.e., while the user is wearing the MR headset.

The portable electronic device 120 may perform operations to identifywhen various rules are satisfied which trigger initiation of the controloperations 404 (FIG. 4) to reduce occurrence of the displayedinformation and the video frames. Various alternative or relatedoperations for triggering the control operations 404 are now explainedin the context of the flowcharts of FIGS. 5 and 6.

Referring to the operations of FIG. 5, the portable electronic device120 determines 500 an amount of overlap that is occurring between: 1)the information that is displayed by the display device and which occursin the video frames from the camera 122; and 2) the locations of thereal-world feature that are identified by processing of the video framesfrom the camera 122. By identifying the overlap between the displayedinformation and the locations of the real-world feature identified inthe video frames, the portable electronic device 120 can determine when,for example, excessive overlap is occurring and responsively initiatethe control operations 404 to reduce such overlap. Thus, the portableelectronic device 120 can initiate 502 the control operations 404 tocontrol at least one of the processing of the video frames and thedisplaying of the information on the display device based on the amountof overlap satisfying a defined rule.

Referring to the alternative or additional operations of FIG. 6, theportable electronic device 120 can display a calibration pattern on thedisplay device, process 602 the video frames to determine a region ofthe video frames containing the graphical pattern, and then cease 604displaying of the calibration pattern on the display device. Followingcessation of display of the calibration pattern, the portable electronicdevice 120 processes 606 the video frames to determine when a locationof the real-world feature enters the region of the video frames.Responsive to the location of the real-world feature entering the regionof the video frames, the portable electronic device 120 initiates 608the operations to control 404 (FIG. 4) at least one of the processing ofthe video frames and the displaying of the information on the displaydevice. Thus, the portable electronic device 120 can selectivelyinitiate the control operations 404 when the MR headset is moved in away that causes a tracked real-world feature to become present in regionof the video frames that has been used to display information forviewing by the user. The control operations 404 may thereby be initiatedonly when needed and, thereby, can avoid possible degradation of thedisplayed information image quality that is viewed by the user and/oravoid possible interference with operation of the processor's ability toidentify locations of the real-world feature in the video frames thatmay otherwise be obscured by the displayed information.

Various operations that can be performed by the portable electronicdevice 120 to reduce occurrence of the displayed information and thevideo frames, are now explained in the context of the alternativeembodiments of the operational flowcharts of FIGS. 7 and 8.

Referring to operations of FIG. 7, the portable electronic device 120can operate to determine 700 when a display quality prioritization ruleis satisfied. Responsive to the display quality prioritization rulebecoming satisfied, the portable electronic device 120 increases 702 afrequency at which the display device is repetitively refreshed todisplay the information and/or decreases 702 a frequency at which thevideo frames from the camera are processed to identify locations of thereal-world feature. Thus, for example, when the user and/or anapplication/operating system executed by the portable electronic device120 activates a mode that prioritizes display quality, the portableelectronic device 120 performs operations that can improve the qualityof the information that is displayed on the display device and projectedon the lens 110 for viewing by the user.

Alternatively or additionally, the portable electronic device 120 canoperate to determine 704 when a camera video prioritization rule issatisfied. Responsive to the camera video prioritization rule becomingsatisfied, the portable electronic device 120 decreases 706 a frequencyat which the display device is repetitively refreshed to display theinformation and/or increases 706 a frequency at which the video framesfrom the camera 122 are processed to identify locations of thereal-world feature. Thus, for example, when the user and/or anapplication/operating system executed by the portable electronic device120 activates a mode that prioritizes camera video quality, the portableelectronic device 120 performs operations that can improve the imagequality of the video frames from the camera 122 to, for example,facilitate more accurate identification of real-world features and/orprovide improved video quality for other use by the portable electronicdevice 120 and/or another electronic device that is networked thereto.

In some alternative or additional embodiments, the portable electronicdevice 120 can at least partially reduce occurrence of displayedinformation in the video frames by controlling the processing the videoframes to only occur while the information is not being displayed on thedisplay device. The portable electronic device 120 may operate torepetitively alternate between displaying the information on the displaydevice and ceasing display of the information on the display device, andcontrolling the processing of the video frames to occur during firsttime durations while the information is not displayed on the displaydevice and to not occur during second time durations while theinformation is displayed on the display device.

Referring to the operations illustrated in FIG. 8, the portableelectronic device 120 determines 800 when a display qualityprioritization rule is satisfied. Responsive to the display qualityprioritization rule becoming satisfied, the portable electronic device120 increases 802 a duty cycle of the second time duration during whichthe display device is repetitively refreshed to display the informationrelative to the first time duration during which the display device isrepetitively refreshed to not display the information. The portableelectronic device 120 may additionally or alternatively determine 804when a camera video prioritization rule is satisfied. Responsive to thecamera video prioritization rule becoming satisfied, portable electronicdevice 120 decreases 806 a duty cycle of the second time duration duringwhich the display device is repetitively refreshed to display theinformation relative to the first time duration during which the displaydevice is repetitively refreshed to not display the information.

Thus, for example, when a mode that prioritizes display quality isactive the portable electronic device 120 can perform operations thatincrease the time duration during which information is displayedrelative to the time duration during which information is not displayed,which can improve the quality of the information that is displayed onthe display device. Alternatively or additionally, when a mode thatprioritizes camera video quality is active the portable electronicdevice 120 can perform operations that decrease the time duration duringwhich information is displayed relative to the time duration duringwhich information is not displayed, which can improve the image qualityof the video frames from the camera 122.

Various of the above operations may be performed by the portableelectronic device 120 to more selectively control the display ofinformation in a region of the display device that would at leastpartially overlap a location of the real-world feature that is to beidentified by the processor. These operations may thereby avoidaffecting how information is displayed in other region(s) of the displaydevice. These and further related operations are now explained in thecontext of the alternative embodiments of the operational flowcharts ofFIGS. 9-12.

Referring to the operations of FIG. 9, the portable electronic device120 processes 900 the video frames from the camera 122 to identify aregion of the display device that displays a portion of informationwhich occurs in the video frames at least partially overlapping alocation of the real-world feature. The portable electronic device 120then repetitively alternates 902 between displaying the portion of theinformation within the region of the display device and ceasing displayof the portion of the information within the region of the displaydevice. The processing of the video frames is controlled to occur duringfirst time durations while the portion of the information is notdisplayed within the region of the display device and controlled to notoccur during second time durations while the portion of the informationis displayed within the region of the display device. Thus, operationsfor alternating between displaying and not displaying the portion of theinformation affects only the region of the display device that has beenidentified as at least partially overlapping a location of thereal-world feature that is to be identified by the processor.

In a further embodiment, the operations to repetitively alternate 902between displaying and ceasing display of the portion of the informationwithin the region of the display device, include determining when adisplay quality prioritization rule is satisfied. Responsive to the rulebecoming satisfied, the portable electronic device 120 increases a dutycycle of the second time duration during which the region of the displaydevice is repetitively refreshed to display the portion of theinformation relative to the second time duration during which the regionof the display device is repetitively refreshed not displaying theportion of the information.

In an alternative further embodiment, the operations to repetitivelyalternate 902 between displaying and ceasing display of the portion ofthe information within the region of the display device, includedetermining when a camera video prioritization rule is satisfied andthen responsive to it becoming satisfied, decreasing a duty cycle of thesecond time duration during which the region of the display device isrepetitively refreshed to display the portion of the informationrelative to the second time duration during which the region of thedisplay device is repetitively refreshed not displaying the portion ofthe information.

The portable electronic device 120 may vary the rate at which twodifferent regions of the video frames are processed to identifyreal-world features based on one of the regions containing informationthat is displayed on the display device and other region not containingthe information. Referring to the example operations illustrated in FIG.10, the portable electronic device 120 processes 1000 the video framesfrom the camera 122 to identify a region of the video frames thatcontains the information. The portable electronic device 120 thenprocesses 1002 the region of the video frames identified as containingthe information at a lower frequency to identify the locations of thereal-world feature than another region of the video frames that does notcontain the information.

The portable electronic device 120 may control the brightness, contrast,and/or color of information for a portion of the display device thatdisplays information which overlaps a location of the real-world featureit is to be identified by the processor. Referring to the exampleoperations illustrated in FIG. 11, the portable electronic device 120processes 1100 the video frames from the camera 122 to identify a regionof the display device that displays a portion of information and whichat least partially overlaps a location of the real-world feature in thevideo frames. The operations to control (404 of FIG. 4) at least one ofthe processing of the video frames and the displaying of the informationon the display device, includes performing 1102 at least one of:decreasing brightness of the region of the display device; decreasingcontrast of the region of the display device; determining a primarycolor of the real-world feature and selecting a color of the informationthat is displayed in the region of the display device to differentiatefrom the primary color of the real-world feature.

Referring to the example further operations illustrated in FIG. 12, theportable electronic device 120 processes 1200 the video frames from thecamera 122 to identify a region of the display device that displays aportion of information and which at least partially overlaps a locationof the real-world feature in the video frames. The operations to control(404 of FIG. 4) at least one of the processing of the video frames andthe displaying of the information on the display device, includesperforming 1202 at least one of: increasing brightness of another regionof the display device that does not overlap a location of the real-worldfeature in the video frames; increasing contrast of the other region ofthe display device; and determining a primary color of the real-worldfeature and selecting a color of the information that is displayed inthe other region of the display device to differentiate from the primarycolor of the real-world feature.

The portable electronic device 120 may identify the region based ondetermining where the user's eyes are presently directed. In oneembodiment, the portable electronic device 120 identifies a region ofthe display device at which the user's eyes are presently directed. Theoperations to control (404 of FIG. 4) at least one of the processing ofthe video frames and the displaying of the information on the displaydevice, includes performing at least one of: decreasing brightness ofthe region of the display device; decreasing contrast of the region ofthe display device; determining a primary color of the real-worldfeature and selecting a color of the information that is displayed inthe region of the display device to differentiate from the primary colorof the real-world feature; increasing brightness of another region ofthe display device that does not overlap a location of the real-worldfeature in the video frames; increasing contrast of the other region ofthe display device; determining a primary color of the real-worldfeature and selecting a color of the information that is displayed inthe other region of the display device to differentiate from the primarycolor of the real-world feature. The portable electronic device 120 mayprocess the video frames from the camera 122 to identify an orientationof a pupil of one of the user's eyes and, therefrom, identify the regionof the display device where the user's eyes are presently directed. In afurther embodiment, the portable electronic device 120 may include aninfrared emitter that illuminates one or both of the pupils of theuser's eyes to enhance their contrast relative to other objects in thecamera's field-of-view and, thereby, facilitate identification of theirorientation relative to the display device.

Cloud Implementation

Some or all operations described above as being performed by a portableelectronic device may alternatively be performed by another node that ispart of a network cloud computing resource. For example, thoseoperations can be performed as a network function by a cloud server or acloud resource of a network operator, such as a gaming or otherentertainment server, map travel routing server, eCommerce server, etc.

Example Portable Electronic Device Configurations

FIG. 13 is a block diagram of components of a portable electronic device120 that are configured to operate in accordance with some otherembodiments of the present disclosure. The portable electronic device120 can include a camera (e.g. a front camera 122 and/or a back camera1250), a display device 1240, at least one processor circuit 1206(processor), and at least one memory 1210 (memory). The portableelectronic device 120 may further include wireless network interfacecircuitry 1230, a motion sensor 1260, and/or a speaker 1270. The motionsensor 1260 may include an inertial measurement unit which outputssignals that are processed by the processor 1206 to determineorientation and/or movement of the portable electronic device 120. Thewireless network interface circuit 1230 may be configured to communicatewith another electronic device through a wireless (e.g., Wi-Fi,Bluetooth, cellular, etc.) network interface.

The processor 1206 is operationally connected to these variouscomponents. The memory 1210 stores program code 1212 that is executed bythe processor 1206 to perform operations. The processor 1206 may includeone or more data processing circuits, such as a general purpose and/orspecial purpose processor (e.g., microprocessor and/or digital signalprocessor), which may be collocated or distributed across one or moredata networks. The processor 1206 is configured to execute computerprogram instructions among program code 1212 in the memory 1210,described below as a computer readable medium, to perform some or all ofthe operations and methods for one or more of the embodiments disclosedherein for a portable electronic device.

Further Definitions and Embodiments

In the above-description of various embodiments of present inventiveconcepts, it is to be understood that the terminology used herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of present inventive concepts. Unless otherwisedefined, all terms (including technical and scientific terms) usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which present inventive concepts belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of this specification andthe relevant art and will not be interpreted in an idealized or overlyformal sense expressly so defined herein.

When an element is referred to as being “connected”, “coupled”,“responsive”, or variants thereof to another element, it can be directlyconnected, coupled, or responsive to the other element or interveningelements may be present. In contrast, when an element is referred to asbeing “directly connected”, “directly coupled”, “directly responsive”,or variants thereof to another element, there are no interveningelements present. Like numbers refer to like elements throughout.Furthermore, “coupled”, “connected”, “responsive”, or variants thereofas used herein may include wirelessly coupled, connected, or responsive.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Well-known functions or constructions may not be described indetail for brevity and/or clarity. The term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third, etc.may be used herein to describe various elements/operations, theseelements/operations should not be limited by these terms. These termsare only used to distinguish one element/operation from anotherelement/operation. Thus, a first element/operation in some embodimentscould be termed a second element/operation in other embodiments withoutdeparting from the teachings of present inventive concepts. The samereference numerals or the same reference designators denote the same orsimilar elements throughout the specification.

As used herein, the terms “comprise”, “comprising”, “comprises”,“include”, “including”, “includes”, “have”, “has”, “having”, or variantsthereof are open-ended, and include one or more stated features,integers, elements, steps, components or functions but does not precludethe presence or addition of one or more other features, integers,elements, steps, components, functions or groups thereof. Furthermore,as used herein, the common abbreviation “e.g.”, which derives from theLatin phrase “exempli gratia,” may be used to introduce or specify ageneral example or examples of a previously mentioned item, and is notintended to be limiting of such item. The common abbreviation “i.e.”,which derives from the Latin phrase “id est,” may be used to specify aparticular item from a more general recitation.

Example embodiments are described herein with reference to blockdiagrams and/or flowchart illustrations of computer-implemented methods,apparatus (systems and/or devices) and/or computer program products. Itis understood that a block of the block diagrams and/or flowchartillustrations, and combinations of blocks in the block diagrams and/orflowchart illustrations, can be implemented by computer programinstructions that are performed by one or more computer circuits. Thesecomputer program instructions may be provided to a processor circuit ofa general purpose computer circuit, special purpose computer circuit,and/or other programmable data processing circuit to produce a machine,such that the instructions, which execute via the processor of thecomputer and/or other programmable data processing apparatus, transformand control transistors, values stored in memory locations, and otherhardware components within such circuitry to implement thefunctions/acts specified in the block diagrams and/or flowchart block orblocks, and thereby create means (functionality) and/or structure forimplementing the functions/acts specified in the block diagrams and/orflowchart block(s).

These computer program instructions may also be stored in a tangiblecomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions whichimplement the functions/acts specified in the block diagrams and/orflowchart block or blocks. Accordingly, embodiments of present inventiveconcepts may be embodied in hardware and/or in software (includingfirmware, resident software, micro-code, etc.) that runs on a processorsuch as a digital signal processor, which may collectively be referredto as “circuitry,” “a module” or variants thereof.

It should also be noted that in some alternate implementations, thefunctions/acts noted in the blocks may occur out of the order noted inthe flowcharts. For example, two blocks shown in succession may in factbe executed substantially concurrently or the blocks may sometimes beexecuted in the reverse order, depending upon the functionality/actsinvolved. Moreover, the functionality of a given block of the flowchartsand/or block diagrams may be separated into multiple blocks and/or thefunctionality of two or more blocks of the flowcharts and/or blockdiagrams may be at least partially integrated. Finally, other blocks maybe added/inserted between the blocks that are illustrated, and/orblocks/operations may be omitted without departing from the scope ofinventive concepts. Moreover, although some of the diagrams includearrows on communication paths to show a primary direction ofcommunication, it is to be understood that communication may occur inthe opposite direction to the depicted arrows.

Many variations and modifications can be made to the embodiments withoutsubstantially departing from the principles of the present inventiveconcepts. All such variations and modifications are intended to beincluded herein within the scope of present inventive concepts.Accordingly, the above disclosed subject matter is to be consideredillustrative, and not restrictive, and the appended examples ofembodiments are intended to cover all such modifications, enhancements,and other embodiments, which fall within the spirit and scope of presentinventive concepts. Thus, to the maximum extent allowed by law, thescope of present inventive concepts are to be determined by the broadestpermissible interpretation of the present disclosure including thefollowing examples of embodiments and their equivalents, and shall notbe restricted or limited by the foregoing detailed description.

1. A portable electronic device for a mixed reality, MR, headsetincluding a lens through which a user wearing the MR headset can viewreal-world features and including a holder configured to releasablyretain the portable electronic device in a defined orientation relativeto the lens, the portable electronic device comprising: a cameraconfigured to output video frames and arranged by the holder to view atleast a portion of the lens of the MR headset; a display device arrangedto display information that is projected on the lens for reflectiondirectly or indirectly toward the user's eyes and the camera of the MRheadset; a processor operationally connected to the camera and to thedisplay device, and configured to perform operations comprising:processing the video frames from the camera to identify locations of atleast one real-world feature; displaying information at locations on thedisplay device; and controlling at least one of the processing of thevideo frames and the displaying of the information on the displaydevice, to at least partially reduce occurrence in the video frames ofthe information that is displayed by the display device.
 2. The portableelectronic device of claim 1, wherein the operations further comprise:determining an amount of overlap occurring between: 1) the informationthat is displayed by the display device and occurring in the videoframes from the camera; and 2) the locations of the real-world featurethat are identified by processing of the video frames from the camera;and initiating the controlling of at least one of the processing of thevideo frames and the displaying of the information on the display devicebased on the amount of overlap satisfying a defined rule.
 3. Theportable electronic device of claim 1, wherein the operations furthercomprise: displaying a calibration pattern on the display device;processing the video frames to determine a region of the video framescontaining the graphical pattern; ceasing display of the calibrationpattern on the display device; following cessation of display of thecalibration pattern, processing the video frames to determine when alocation of the real-world feature enters the region of the videoframes; and responsive to the location of the real-world featureentering the region of the video frames, initiating the controlling ofat least one of the processing of the video frames and the displaying ofthe information on the display device.
 4. The portable electronic deviceof claim 1, wherein the operations to control at least one of theprocessing of the video frames and the displaying of the information onthe display device, further comprise: determining when a display qualityprioritization rule is satisfied; and responsive to the display qualityprioritization rule becoming satisfied, increasing a frequency at whichthe display device is repetitively refreshed to display the informationand/or decreasing a frequency at which the video frames from the cameraare processed to identify locations of the real-world feature.
 5. Theportable electronic device of claim 1, wherein the operations to controlat least one of the processing of the video frames and the displaying ofthe information on the display device, further comprise: determiningwhen a camera video prioritization rule is satisfied; and responsive tothe camera video prioritization rule becoming satisfied, decreasing afrequency at which the display device is repetitively refreshed todisplay the information and/or increasing a frequency at which the videoframes from the camera are processed to identify locations of thereal-world feature.
 6. The portable electronic device of claim 1,wherein the operations to control at least one of the processing of thevideo frames and the displaying of the information on the displaydevice, further comprise: repetitively alternating between displayingthe information on the display device and ceasing display of theinformation on the display device, wherein the processing of the videoframes is controlled to occur during first time durations while theinformation is not displayed on the display device and controlled to notoccur during second time durations while the information is displayed onthe display device.
 7. The portable electronic device of claim 6,wherein the operations to repetitively alternate between displaying theinformation on the display device and ceasing display of the informationon the display device, further comprise: determining when a displayquality prioritization rule is satisfied; and responsive to the displayquality prioritization rule becoming satisfied, increasing a duty cycleof the second time duration during which the display device isrepetitively refreshed to display the information relative to the firsttime duration during which the display device is repetitively refreshednot displaying the information.
 8. The portable electronic device ofclaim 6, wherein the operations to repetitively alternate betweendisplaying the information on the display device and ceasing display ofthe information on the display device, further comprise: determiningwhen a camera video prioritization rule is satisfied; and responsive tothe camera video prioritization rule becoming satisfied, decreasing aduty cycle of the second time duration during which the display deviceis repetitively refreshed to display the information relative to thefirst time duration during which the display device is repetitivelyrefreshed not displaying the information.
 9. The portable electronicdevice of claim 1, wherein: the operations further comprise processingthe video frames from the camera to identify a region of the displaydevice that displays a portion of information which occurs in the videoframes at least partially overlapping a location of the real-worldfeature, and the operations to control at least one of the processing ofthe video frames and the displaying of the information on the displaydevice, further comprise: repetitively alternating between displayingthe portion of the information within the region of the display deviceand ceasing display of the portion of the information within the regionof the display device, wherein the processing of the video frames iscontrolled to occur during first time durations while the portion of theinformation is not displayed within the region of the display device andcontrolled to not occur during second time durations while the portionof the information is displayed within the region of the display device.10. The portable electronic device of claim 9, wherein the operations torepetitively alternate between displaying the portion of the informationwithin the region of the display device and ceasing display of theportion of the information within the region of the display device,further comprise: determining when a display quality prioritization ruleis satisfied; and responsive to the display quality prioritization rulebecoming satisfied, increasing a duty cycle of the second time durationduring which the region of the display device is repetitively refreshedto display the portion of the information relative to the second timeduration during which the region of the display device is repetitivelyrefreshed not displaying the portion of the information.
 11. Theportable electronic device of claim 9, wherein the operations torepetitively alternate between displaying the portion of the informationwithin the region of the display device and ceasing display of theportion of the information within the region of the display device,further comprise: determining when a camera video prioritization rule issatisfied; and responsive to the camera video prioritization rulebecoming satisfied, decreasing a duty cycle of the second time durationduring which the region of the display device is repetitively refreshedto display the portion of the information relative to the second timeduration during which the region of the display device is repetitivelyrefreshed not displaying the portion of the information.
 12. Theportable electronic device of claim 1, wherein: the operations furthercomprise processing the video frames from the camera to identify aregion of the video frames that contains the information; and theoperations to control at least one of the processing of the video framesand the displaying of the information on the display device, furthercomprise: processing the region of the video frames identified ascontaining the information at a lower frequency than another region ofthe video frames that does not contain the information, to identify thelocations of the real-world feature.
 13. The portable electronic deviceof claim 1, wherein: the operations further comprise processing thevideo frames from the camera to identify a region of the display devicethat displays a portion of information and which at least partiallyoverlaps a location of the real-world feature in the video frames, andthe operations to control at least one of the processing of the videoframes and the displaying of the information on the display device,further comprise: performing at least one of: decreasing brightness ofthe region of the display device; decreasing contrast of the region ofthe display device; determining a primary color of the real-worldfeature and selecting a color of the information that is displayed inthe region of the display device to differentiate from the primary colorof the real-world feature.
 14. The portable electronic device of claim1, wherein: the operations further comprise processing the video framesfrom the camera to identify a region of the display device that displaysa portion of information and which at least partially overlaps alocation of the real-world feature in the video frames, and theoperations to control at least one of the processing of the video framesand the displaying of the information on the display device, furthercomprise: performing at least one of: increasing brightness of anotherregion of the display device that does not overlap a location of thereal-world feature in the video frames; increasing contrast of the otherregion of the display device; and determining a primary color of thereal-world feature and selecting a color of the information that isdisplayed in the other region of the display device to differentiatefrom the primary color of the real-world feature.
 15. The portableelectronic device of claim 1, wherein: the operations further compriseidentifying a region of the display device at which the user's eyes arepresently directed, and the operations to control at least one of theprocessing of the video frames and the displaying of the information onthe display device, further comprise: performing at least one of:decreasing brightness of the region of the display device; decreasingcontrast of the region of the display device; determining a primarycolor of the real-world feature and selecting a color of the informationthat is displayed in the region of the display device to differentiatefrom the primary color of the real-world feature; increasing brightnessof another region of the display device that does not overlap a locationof the real-world feature in the video frames; increasing contrast ofthe other region of the display device; determining a primary color ofthe real-world feature and selecting a color of the information that isdisplayed in the other region of the display device to differentiatefrom the primary color of the real-world feature.
 16. A method by aportable electronic device for a mixed reality, MR, headset including alens through which a user wearing the MR headset can view real-worldfeatures and including a holder configured to releasably retain theportable electronic device in a defined orientation relative to thelens, wherein a display device of the portable electronic device isarranged to display information that is projected on the lens forreflection directly or indirectly toward the user's eyes and a camera ofthe portable electronic device, the method comprising: processing thevideo frames from the camera of the portable electronic device toidentify locations of at least one real-world feature; displayinginformation at locations on the display device of the portableelectronic device; and controlling at least one of the processing of thevideo frames and the displaying of the information on the displaydevice, to at least partially reduce occurrence in the video frames ofthe information that is displayed by the display device.
 17. The methodof claim 16, further comprising: determining an amount of overlapoccurring between: 1) the information that is displayed by the displaydevice and occurring in the video frames from the camera; and 2) thelocations of the real-world feature that are identified by processing ofthe video frames from the camera; and initiating the controlling of atleast one of the processing of the video frames and the displaying ofthe information on the display device based on the amount of overlapsatisfying a defined rule.
 18. The method of any of claim 16, furthercomprising; displaying a calibration pattern on the display device;processing the video frames to determine a region of the video framescontaining the graphical pattern; ceasing display of the calibrationpattern on the display device; following cessation of display of thecalibration pattern, processing the video frames to determine when alocation of the real-world feature enters the region of the videoframes; and responsive to the location of the real-world featureentering the region of the video frames, initiating the controlling ofat least one of the processing of the video frames and the displaying ofthe information on the display device.
 19. The method of claim 16,wherein controlling at least one of the processing of the video framesand the displaying of the information on the display device, furthercomprises: determining when a display quality prioritization rule issatisfied; and responsive to the display quality prioritization rulebecoming satisfied, increasing a frequency at which the display deviceis repetitively refreshed to display the information and/or decreasing afrequency at which the video frames from the camera are processed toidentify locations of the real-world feature.
 20. The method of claim16, wherein controlling at least one of the processing of the videoframes and the displaying of the information on the display device,further comprises: determining when a camera video prioritization ruleis satisfied; and responsive to the camera video prioritization rulebecoming satisfied, decreasing a frequency at which the display deviceis repetitively refreshed to display the information and/or increasing afrequency at which the video frames from the camera are processed toidentify locations of the real-world feature.
 21. The method of claim16, wherein controlling at least one of the processing of the videoframes and the displaying of the information on the display device,further comprises: repetitively alternating between displaying theinformation on the display device and ceasing display of the informationon the display device, wherein the processing of the video frames iscontrolled to occur during first time durations while the information isnot displayed on the display device and controlled to not occur duringsecond time durations while the information is displayed on the displaydevice.
 22. The method of claim 21, wherein repetitively alternatingbetween displaying the information on the display device and ceasingdisplay of the information on the display device, further comprises:determining when a display quality prioritization rule is satisfied; andresponsive to the display quality prioritization rule becomingsatisfied, increasing a duty cycle of the second time duration duringwhich the display device is repetitively refreshed to display theinformation relative to the first time duration during which the displaydevice is repetitively refreshed not displaying the information.
 23. Themethod of claim 21, wherein repetitively alternating between displayingthe information on the display device and ceasing display of theinformation on the display device, further comprises: determining when acamera video prioritization rule is satisfied; and responsive to thecamera video prioritization rule becoming satisfied, decreasing a dutycycle of the second time duration during which the display device isrepetitively refreshed to display the information relative to the firsttime duration during which the display device is repetitively refreshednot displaying the information.
 24. The method of claim 16, furthercomprising processing the video frames from the camera to identify aregion of the display device that displays a portion of informationwhich occurs in the video frames at least partially overlapping alocation of the real-world feature, and wherein controlling at least oneof the processing of the video frames and the displaying of theinformation on the display device, further comprises: repetitivelyalternating between displaying the portion of the information within theregion of the display device and ceasing display of the portion of theinformation within the region of the display device, wherein theprocessing of the video frames is controlled to occur during first timedurations while the portion of the information is not displayed withinthe region of the display device and controlled to not occur duringsecond time durations while the portion of the information is displayedwithin the region of the display device.
 25. The method of claim 24,wherein repetitively alternating between displaying the portion of theinformation within the region of the display device and ceasing displayof the portion of the information within the region of the displaydevice, further comprises: determining when a display qualityprioritization rule is satisfied; and responsive to the display qualityprioritization rule becoming satisfied, increasing a duty cycle of thesecond time duration during which the region of the display device isrepetitively refreshed to display the portion of the informationrelative to the second time duration during which the region of thedisplay device is repetitively refreshed not displaying the portion ofthe information.
 26. The method of claim 24, wherein repetitivelyalternating between displaying the portion of the information within theregion of the display device and ceasing display of the portion of theinformation within the region of the display device, further comprises:determining when a camera video prioritization rule is satisfied; andresponsive to the camera video prioritization rule becoming satisfied,decreasing a duty cycle of the second time duration during which theregion of the display device is repetitively refreshed to display theportion of the information relative to the second time duration duringwhich the region of the display device is repetitively refreshed notdisplaying the portion of the information.
 27. The method of claim 16,further comprising processing the video frames from the camera toidentify a region of the video frames that contains the information, andwherein controlling at least one of the processing of the video framesand the displaying of the information on the display device, furthercomprises: processing the region of the video frames identified ascontaining the information at a lower frequency than another region ofthe video frames that does not contain the information, to identify thelocations of the real-world feature.
 28. The method of claim 16, furthercomprising processing the video frames from the camera to identify aregion of the display device that displays a portion of informationwhich at least partially overlaps a location of the real-world featurein the video frames, and wherein controlling at least one of theprocessing of the video frames and the displaying of the information onthe display device, further comprises: performing at least one of:decreasing brightness of the region of the display device; decreasingcontrast of the region of the display device; and determining a primarycolor of the real-world feature and selecting a color of the informationthat is displayed in the region of the display device to differentiatefrom the primary color of the real-world feature.
 29. The method ofclaim 16, further comprising processing the video frames from the camerato identify a region of the display device that displays a portion ofinformation which at least partially overlaps a location of thereal-world feature in the video frames, and wherein controlling at leastone of the processing of the video frames and the displaying of theinformation on the display device, further comprises: performing atleast one of: increasing brightness of another region of the displaydevice that does not overlap a location of the real-world feature in thevideo frames; increasing contrast of the other region of the displaydevice; determining a primary color of the real-world feature andselecting a color of the information that is displayed in the otherregion of the display device to differentiate from the primary color ofthe real-world feature.
 30. The method of claim 16: further comprisingidentifying a region of the display device at which the user's eyes arepresently directed, and wherein controlling at least one of theprocessing of the video frames and the displaying of the information onthe display device, further comprises: performing at least one of:decreasing brightness of the region of the display device; decreasingcontrast of the region of the display device; determining a primarycolor of the real-world feature and selecting a color of the informationthat is displayed in the region of the display device to differentiatefrom the primary color of the real-world feature; increasing brightnessof another region of the display device that does not overlap a locationof the real-world feature in the video frames; increasing contrast ofthe other region of the display device; determining a primary color ofthe real-world feature and selecting a color of the information that isdisplayed in the other region of the display device to differentiatefrom the primary color of the real-world feature.
 31. A computer programproduct for a portable electronic device for a mixed reality, MR,headset having a lens through which a user wearing the MR headset canview real-world features, wherein a display device of the portableelectronic device is arranged to display information that is projectedon the lens for reflection directly or indirectly toward the user's eyesand a camera of the portable electronic device, the computer programproduct comprising: a non-transitory computer readable medium storingprogram code executable by a processor of the portable electronic deviceto perform operations comprising: processing the video frames from thecamera of the portable electronic device to identify locations of atleast one real-world feature; displaying information at locations on thedisplay device of the portable electronic device; and controlling atleast one of the processing of the video frames and the displaying ofthe information on the display device, to at least partially reduceoccurrence in the video frames of the information that is displayed bythe display device.
 32. (canceled)